cleanup processed files in VR by DaanHoogland · Pull Request #7632 · apache/cloudstack

DaanHoogland · 2023-06-15T06:45:38Z

Description

When a PR is long running, the directory /var/cache/cloud/processed filles up and might glock the filesystem, rendring the VR unusable.
This PR implements a mech for cleaning that directory on demand.
(to be triggered yet, by API/schedule/configurable timer???)

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

codecov · 2023-06-19T14:58:06Z

Codecov Report

Merging #7632 (38089c7) into 4.18 (e4117e6) will decrease coverage by 0.04%.
The diff coverage is 54.76%.

❗ Current head 38089c7 differs from pull request most recent head 555e054. Consider uploading reports for the commit 555e054 to get more accurate results

@@             Coverage Diff              @@
##               4.18    #7632      +/-   ##
============================================
- Coverage     13.05%   13.02%   -0.04%     
+ Complexity     9083     9032      -51     
============================================
  Files          2720     2720              
  Lines        257370   257067     -303     
  Branches      40124    40087      -37     
============================================
- Hits          33612    33476     -136     
+ Misses       219537   219387     -150     
+ Partials       4221     4204      -17

Files Changed	Coverage Δ
...va/com/cloud/agent/manager/DirectAgentAttache.java	`9.83% <0.00%> (ø)`
...tack/engine/orchestration/NetworkOrchestrator.java	`6.06% <0.00%> (ø)`
...ork/router/VirtualNetworkApplianceManagerImpl.java	`2.72% <0.00%> (-0.01%)`	⬇️
...network/router/VirtualNetworkApplianceManager.java	`100.00% <100.00%> (ø)`

... and 29 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

DaanHoogland · 2023-07-14T09:40:04Z

core/src/main/java/com/cloud/agent/api/CleanProcessedCacheCommand.java

+
+import com.cloud.agent.api.routing.NetworkElementCommand;
+
+public class CleanProcessedCacheCommand extends NetworkElementCommand {


nts: does this mean the answer should also inherit (from NetworkElementAnswer for instance)

DaanHoogland · 2023-08-07T08:16:04Z

@blueorangutan package

blueorangutan · 2023-08-07T08:18:05Z

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-07T09:28:03Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6690

DaanHoogland · 2023-08-07T09:38:11Z

@blueorangutan test keepEnv

blueorangutan · 2023-08-07T09:40:04Z

@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-08-07T20:56:10Z

[SF] Trillian test result (tid-7316)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39009 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7632-t7316-kvm-centos7.zip
Smoke tests completed. 95 look OK, 13 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_network_acl	`Error`	39.99	test_network_acl.py
test_01_vpc_privategw_acl	`Failure`	42.66	test_privategw_acl.py
test_02_vpc_privategw_static_routes	`Failure`	39.65	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	42.62	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	46.12	test_privategw_acl.py
test_10_vpc_tier_kubernetes_cluster	`Error`	82.22	test_kubernetes_clusters.py
test_02_native_to_native_vpc_migration	`Error`	40.80	test_migration.py
test_03_create_vpc_domain_vpc_offering	`Error`	51.28	test_domain_vpc_offerings.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	`Failure`	46.52	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Failure`	46.54	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	`Failure`	44.63	test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Failure`	46.49	test_internal_lb.py
test_01_single_VPC_iptables_policies	`Error`	41.16	test_routers_iptables_default_policy.py
test_07_autoscaling_vmgroup_on_vpc_network	`Error`	43.05	test_vm_autoscaling.py
test_01_migrate_VM_and_root_volume	`Error`	81.75	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	53.38	test_vm_life_cycle.py
test_01_verify_ipv6_vpc	`Error`	41.15	test_vpc_ipv6.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	41.79	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Error`	42.82	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Error`	45.82	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	43.77	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	41.72	test_vpc_redundant.py
test_01_VPC_nics_after_destroy	`Error`	39.56	test_vpc_router_nics.py
test_02_VPC_default_routes	`Error`	40.68	test_vpc_router_nics.py
test_01_redundant_vpc_site2site_vpn	`Failure`	42.20	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Failure`	43.21	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	`Failure`	34.87	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Failure`	44.36	test_vpc_vpn.py

DaanHoogland · 2023-08-10T09:43:09Z

vpc creation failes after these changes: needs work

weizhouapache · 2023-08-21T06:40:54Z

@DaanHoogland
there was a commit for similar issue see #5331
the script runs once every day. we could let it run every hour or every minute.

DaanHoogland · 2023-08-23T09:07:58Z

@DaanHoogland there was a commit for similar issue see #5331 the script runs once every day. we could let it run every hour or every minute.

let's push this forward then (marking 4.18.2 for now and we can discuss if it is still applicable)

…g service

DaanHoogland · 2023-10-02T13:47:44Z

@blueorangutan LLpackage

blueorangutan · 2023-10-02T13:48:03Z

@DaanHoogland a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-10-02T14:34:59Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6191

DaanHoogland · 2023-10-03T06:58:00Z

@blueorangutan LLtest matrix

blueorangutan · 2023-10-03T07:00:03Z

@DaanHoogland a [LL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-10-03T07:18:23Z

[LL] Trillian Build Failed (tid-6763)

DaanHoogland · 2023-10-06T07:54:10Z

no interest

boring-cyborg bot added component:api component:networking component:virtual-router component:agent component:orchestration Python Warning... Python code Ahead! labels Jun 15, 2023

DaanHoogland changed the title ~~cleanup processed in VR~~ cleanup processed files in VR Jun 19, 2023

apache deleted a comment from blueorangutan Jun 19, 2023

weizhouapache added this to the 4.18.1.0 milestone Jun 20, 2023

weizhouapache added the Severity:Minor label Jun 26, 2023

apache deleted a comment from blueorangutan Jul 11, 2023

DaanHoogland commented Jul 14, 2023

View reviewed changes

apache deleted a comment from blueorangutan Jul 18, 2023

DaanHoogland force-pushed the logRotateAndSuch branch 2 times, most recently from 4792a6c to 1563e28 Compare July 26, 2023 07:13

DaanHoogland force-pushed the logRotateAndSuch branch from 1563e28 to 28db7f8 Compare August 4, 2023 13:46

apache deleted a comment from blueorangutan Aug 4, 2023

DaanHoogland force-pushed the logRotateAndSuch branch from 28db7f8 to 38089c7 Compare August 7, 2023 08:15

apache deleted a comment from blueorangutan Aug 7, 2023

DaanHoogland force-pushed the logRotateAndSuch branch from 38089c7 to 38c6f03 Compare August 10, 2023 12:58

DaanHoogland modified the milestones: 4.18.1.0, 4.18.2.0 Aug 23, 2023

cleanup proccessed files in VR by setting cron on starting monitorrin…

555e054

…g service

DaanHoogland force-pushed the logRotateAndSuch branch from 38c6f03 to 555e054 Compare August 23, 2023 09:08

DaanHoogland closed this Oct 6, 2023


		import com.cloud.agent.api.routing.NetworkElementCommand;

		public class CleanProcessedCacheCommand extends NetworkElementCommand {

Conversation

DaanHoogland commented Jun 15, 2023

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov bot commented Jun 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland Jul 14, 2023

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Aug 7, 2023

Uh oh!

blueorangutan commented Aug 7, 2023

Uh oh!

blueorangutan commented Aug 7, 2023

Uh oh!

DaanHoogland commented Aug 7, 2023

Uh oh!

blueorangutan commented Aug 7, 2023

Uh oh!

blueorangutan commented Aug 7, 2023

Uh oh!

DaanHoogland commented Aug 10, 2023

Uh oh!

weizhouapache commented Aug 21, 2023

Uh oh!

DaanHoogland commented Aug 23, 2023

Uh oh!

DaanHoogland commented Oct 2, 2023

Uh oh!

blueorangutan commented Oct 2, 2023

Uh oh!

blueorangutan commented Oct 2, 2023

Uh oh!

DaanHoogland commented Oct 3, 2023

Uh oh!

blueorangutan commented Oct 3, 2023

Uh oh!

blueorangutan commented Oct 3, 2023

Uh oh!

DaanHoogland commented Oct 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Jun 19, 2023 •

edited

Loading